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Using Technology in Helping Students Achieve 21 st Century Skills: 

A Pilot Study 

Background 

As we enter the 21 st Century there is a great deal of discussion in 
business and education circles alike about the type of skills our youth will need 
to survive and thrive in this century. At the same time, there is little known 
today about the level of 21 st Century skills students currently have. In part this 
is because, as a nation, we are still in the process of articulating the specific 
skills needed, and in part it is because too little systemic examination of 
existing skill levels has been undertaken. Educational Testing Service (ETS) 
has begun to address this issue by developing a 75-minute scenario-based test 
to measure high school senior and college freshmen students’ Information and 
Communication Technology (ICT) Literacy skills; skills defined by ETS as, “the 
ability of post-secondary students to: define, access, manage, integrate, 
evaluate, create, and communicate information in a technological environment,” 
(http: / / theiournal.com/articles/ 17084) . 

During the spring of 2006, ETS offered high schools and universities 
across the country the opportunity to take an early version of the assessment. 
One of those high schools was Skowhegan Area High School (SAHS) in Maine 
School Administrative District #54 (MSAD #54). A total of 279 students (70 
ninth graders, 26 tenth graders, 162 twelfth graders and 21 “others”) took the 
ETS Early 2006 Administration Core Level test along with 658 high school 
seniors from eight other high schools across the country. An additional 2,559 
college students at various universities also took the ICT exam. 

Skowhegan Area High School Performance on the ETS ICT Assessment 

An analysis of the test results for Skowhegan Area High School revealed 
that the schools’ students performed well on the assessment when compared 
with other students participating in the April 2006 testing. As may be seen in 
Table 1 on the next page, despite the fact that the reading level of the ICT 
assessment was targeted for high school senior/ college freshmen and the 
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economic advantages of the majority of participating high schools when 
compared with SAHS (see Appendix A for comparison list of schools), 
Skowhegan freshmen scored slightly above all other high school seniors taking 
the exam (127.3 mean score vs. 126.6). In addition, the Skowhegan seniors 
planning on going to a four year college scored better than their counterparts 
at the other high schools (149.7 mean score vs. 132.6). 



Table 1: SAHS and other Schools’ Students ETS Test Scores 





SAHS Freshmen 


SAHS Seniors 


ETS HS Seniors 
(excludes SAHS) 


n 


mean 


std. dev. 


n 


mean 


std. dev. 


n 


mean 


std. dev. 


All Students 


70 


127.3 


32.65 


162 


130.1 


39.05 


658 


126.6 


NA 


Not College Bound 


** 


** 


** 


38 


103 


26.977 


68 


104.5 


NA 


2 Yr. College 


** 


** 


** 


49 


120.5 


37.308 


94 


111.9 


NA 


4 Yr. College 


48 


132.6 


33.15 


72 


149.7 


35.48 


499 


132.6 


NA 



** base sizes too small 



The results suggested that the work Skowhegan has been doing 
preparing students for the 21 st Century is showing some progress. It must be 
noted, however, that the large variance associated with student scores in this 
ETS version of the assessment (measured in standard deviations) suggests 
some limitations of the ETS assessment and/or considerable differences in 
student performance on the test. Still, even with these possible limitations the 
results are promising. 

Further analysis of the ETS test results by SAHS staff with help from the 
Center for Education Policy, Applied Research, and Evaluation (CEPARE) at the 
University of Southern Maine (USM), indicated that students at Skowhegan 
were fairly skilled at locating potentially useful websites for school related 
work, but could benefit from instruction in four of the tested skill areas 
(evaluate, integrate, create and communicate). The SAHS team of 
administrators, technology integrationists, curriculum specialists, and teachers 
decided to focus their effort on helping students enhance their skills in the 
specific area of evaluating information obtained online. According to ETS 
documents, this includes the ability to judge the quality, relevance, authority, 
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point of view/ bias, currency, coverage or accuracy of digital information. At 
this point CEPARE formed a partnership with MSAD #54 and SAHS to develop 
a model/ process by which 21 st Century Skills could be taught to students. 
School-based Need 

As a result of the ETS exam, school officials were made more aware of 
the lack of continuity in the teaching of research skills amongst classrooms 
throughout the school. They agreed that working with CEPARE on this project 
would allow them the opportunity to create and test materials that could 
potentially be given to all teachers for use on all assigned research projects. 
This type of cross-curricular tool would allow students access to the same 
process in multiple subjects thus increasing the likelihood that transference 
would take place. The high school principal noted that communication 
between the computer applications teachers, who are responsible for teaching 
technological and citation skills as they relate to research, and the 
English/ Language Arts teachers, who are responsible for using that format in 
the classroom is poor. In addition, there is little communication between other 
subject area teachers and those teachers responsible for teaching research 
skills. The ETS exam results indicate that some of what is happening in 
Skowhegan is working quite well, but school officials and CEPARE staff agreed 
that reinforcing these skills in an organized way would improve the students’ 
ability to conduct online research even more. 

Methodology 

Initial project planning meetings took place during November and 
December of 2006. During those meetings, participating staff and teachers 
were brought together to discuss and plan the project. The team made the 
decision to include upper elementary students (6 th grade) as well as middle 
school students (7 th and 8 th grades) in addition to 9 th grade students at SAHS 
in the project. The ability to effectively evaluate information obtained online is 
important for all students doing research and is particularly important in 
middle schools in Maine where all 7 th and 8 th grade students learn in a 1:1 
laptop environment. 
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Goals of the Project 

The primary goal of this collaborative project between MSAD #54 and 
CEPARE was to create a model/ process to help students in 6 th , 7 th , 8 th , and 9 th 
grades learn how to evaluate electronic/ digital resources within the context of 
authentic learning activities. 

The team set the following guidelines for the project: 

1. Input to the project should be as broad as possible and include 
expertise from teachers, administrators, technology integrationists 
as well as curriculum specialists. 

2. Materials developed (the “intervention”) should be designed for use 
by classroom teachers and be integrated into the existing 
curriculum. 

3. Teachers participating in the project should receive a small stipend 
for the extra work required in conducting the study (collecting 
student data, providing researchers with feedback and 
documentation, attending meetings). 

The team set a time frame to conduct the pilot project/ research with students 
during April and May of 2007. The intervening time between November 2006 
and April 2007 was used to develop a project plan, create materials for the 
intervention, create teacher and student interview guides, and to develop 
assessment tools. A more detailed project task list and timeline appears in 
Appendix B. 

Project Staff 

The project was spearheaded at Skowhegan primarily by Dorothy Small, 
Technology Integrationist at SAHS and Rick Wilson, principal at SAHS. Both 
Small and Wilson worked together to coordinate meetings, communicate with 
teachers, and serve as the link between CEPARE and MSAD #54. In addition 
to these two key players, several others played important roles in providing 
support for the project. These included: 
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Participant 


Role 


Laura Richter 


Technology Integrationist, Skowhegan Area Middle School (SAMS) 


Steve Chiasson 


Technology Integrationist, K-6 


John Krasnavage 


Principal. Skowhegan Area Middle School (SAMS) 


Dawnela Sheehan 


Assessment Coordinator, SAD #54 


Erin Wood 


7-12 Literacy Specialist 


Marti Pakulski 


Library Media Director, SAHS 


Sharon Lambert 


Technology & Career Education Department, SAHS 


Leanne Walker 


Research Analyst, CEP ARE 


Sarah Wintle 


Research Associate, CEP ARE 


David Silvernail 


Director, CEP ARE 



In addition, six teachers from grades 6, 7, 8 8s 9 were recruited to 
participate as either control or experimental teachers. Their primary roles were 
to implement the curriculum intervention materials with students in their 
classrooms and administer pre-and post assessments. Experimental and 
control classroom teachers working on the project included: 





Experimental Group 


Control Group 


Grade 6 

(Margaret Chase Smith Elem. School) 


Linda Greenlaw 


Mary Herrick 


Grades 7 and 8 

(Skowhegan Area Middle School) 


Julie Wallace, Science 


Jason Bellerose, Social Studies 


Grade 9 

(Skowhegan Area High School) 


Kate Drummond, Science 


Scott Pillsbury, Science 



Intervention 



The curriculum intervention materials were created primarily by the 
technology integrationists, Dorothy Small, Laura Richter, and Steve Chiasson, 
on the MSAD #54 team. The materials were designed to be used by each of the 
experimental teachers, irrespective of the grade level and content currently 
being taught. A major goal of the project overall was to create a process that 
could be translated to various grade levels and across all subjects so that 
ultimately learning could be reinforced throughout the students’ academic 
experience. The intervention consisted of a Teacher Guide as well as a Student 
Resource Guide (see Appendix C for a copy of the materials). 
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The intervention focused on enabling students to gain skills in answering 
three key questions: Does the content of the website appear to be useful? 

What is the apparent purpose of the website? How reliable is the information 
contained on the website? The amount of time teachers spent providing the 
intervention to their students was determined by the teachers themselves and 
varied among teachers and grade levels. No guidelines were specified by the 
project team and teachers were encouraged to use the materials in whatever 
content area they deemed appropriate. The experimental classroom teachers 
reported spending a total of between 45-60 minutes of instruction in 6 th , 30 
minutes of instruction in 7 th 8& 8 th grades and two hours of instruction in 9 th 
grade. 

Assessments 

The assessment was scenario based (students were asked to plan a 
week’s worth of healthy menus by seeking out information online using three 
pre-determined web sites). Students were directed to three specific websites 
and were then asked to evaluate the usefulness, relevance, purpose, and 
reliability of the three websites in relation to the task they had been given. A 
copy of the assessment and websites used in conjunction with that document 
appear in Appendices D and E. 

The assessments were developed by the CEPARE and MSAD #54 team, 
and pre-tested for appropriateness and clarity. Several students were asked to 
take the draft assessment and were interviewed by the SAD #54 technology 
integrationists to check for language difficulties and clarity of instructions. As 
a result of this student input, slight wording changes were made to the final 
version of the assessment. 

Once the assessment was finalized, control and experimental students 
completed the online assessment between April 6 th and April 10 th . The same 
assessment was used as the post-test for both the experimental and control 
groups after the intervention had been administered. The post assessment was 
taken over a period of 6 weeks because the intervention was designed to be 
used within the context of the participating teacher’s curriculum and specific 
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dates for the instruction could not be set. In order to develop a scoring rubric 
for the assessments, the MSAD #54 technology integrationists who created the 
intervention were asked to take the assessment online. Their responses were 
used by CEPARE to develop the scoring rubric (see Appendix H). 

Student sample sizes for each grade are included in the table below: 



Table 2: Student Participation - Experimental vs. Control by Grade 





6th grade 


7th grade 


8th grade 


9th grade 


Total 


Pre Assessment 


n= 


n= 


n= 


n= 


n= 


Experimental 


16 


18 


15 


23 


72 


Control 


15 


19 


19 


65 


118 


Post Assessment 


n= 


n= 


n= 


n= 


n= 


Experimental 


16 


18 


15 


18 


67 


Control 


15 


18 


17 


51 


101 


Matched Pre-Post 


n= 


n= 


n= 


n= 


n= 


Experimental 


16 


18 


15 


18 


67 


j Control 


15 


18 


17 


51 


101 



The pre-and post-assessments were scored by the CEPARE project staff. 
Careful calibration of the rubric was done by CEPARE staff members on a 
sample of student assessments at the start of the scoring process as well as 
several times during scoring of the approximately 350 pre-and post- 
assessments. In order to score each exam individually, student assessments 
were grouped randomly into sets of 50-75. Three members of the research staff 
scored one set of student responses at a time, meeting after each set to 
determine if exams were being scored consistently. Student scores were based 
on values assigned using the rubric as a guide. 

Student Interviews 

Twelve students were interviewed by CEPARE staff. Four students from 
each grade level (two experimental and two controls) were interviewed after 
taking the pre-assessment. The same students were interviewed again after 
taking the post-assessment. The purpose of these interviews was to better 
understand the student’s thought processes involved when they are evaluating 
a website as well as to obtain feedback on the intervention, assessments and 
project overall. Student interview questions appear in Appendix F. 
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Teacher Interviews 

Rick Wilson, SAHS principal, interviewed all participating classroom 
teachers both prior to and after completion of the project. Teachers were asked 
questions about previous student work around evaluation skills as well as 
about the presupposed skills of students currently in their classes. Teacher 
interview questions appear in Appendix G. 

Results 

Results from the pre-and post-assessments were analyzed using 
descriptive and inferential statistics. Analysis of the pre- and post-assessment 
scores indicated that the scores of students who received the intervention were 
significantly higher on the post-assessment than scores of students who did 
not receive the intervention for the 7 th , 8 th and 9 th grade students combined. 
This information appears in Table 3. Further analysis indicated there were no 
statistically significant differences in pre-test scores for the experimental and 
control groups (p>.05), but there were significant post-test differences (p<.05). 
The experimental group students outperformed their control group cohorts. In 
addition, analysis of the data for different grade levels indicated that the 
intervention was most effective with 8th graders, and somewhat mixed for the 
other grade levels. 



Table 3: Pre-Post Assessment Differences - Experimental vs. Control Groups 



7 th / 8 th /9 th Grades 


Mean Std. Dev. P value Effect Size 


Exp. 

Pre Test 

Control 


14.55 4.49 0.262 0.19 

15.52 5.11 




7 th / 8 th /9 th Grades 


Mean Std. Dev. P value Effect Size 


Exp. 

Post Test 

Control 


16.47 5.5 0.021 0.409 

14.19 5.58 



It should be noted that sixth grade students were excluded from the analysis. 
This was done because the interview evidence indicated that 6 th graders 
struggled with the intervention. The one-on-one interviews revealed a great 
deal of confusion about the intervention among the four students interviewed. 
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The students did not appear to understand that they were being asked to 
evaluate three different web sites looking for information that would help them 
prepare a week’s worth of menus. They were also unable to explain or 
elaborate on the responses they had given on the assessments. 

An analysis of student responses to individual assessment questions also 
revealed mixed results. Table 4 reports the questions that were asked about 
each of three web sites as well as the total number of possible points students 
could have received on the post-assessment (among all students who took the 
post-assessment), the number of actual points received, and the percentage of 
correct answers those points represent. 

The majority of students (57%) were able to determine how useful a 
website would be to them when given an assignment and were able to 
determine the main purpose of the website. The majority (58%) were also able 
to distinguish fact from opinion. Only 25%, however, were able to correctly 
identify a website as being a primary or secondary source. Many students 
confused the word “primary” with the words “most important or main” in the 
one-on-one interviews. 

Students also had trouble explaining why they selected a particular 
response. Fewer than a third (27%) were able to support their reasoning in 
selecting the main purpose of a website and only 1 5% were able to explain why 
they considered information on a website to be either fact or opinion. Students 
also had trouble identifying the author or sponsor of a particular website (only 
29% could do this). They were also unable to list sources referenced in a 
website. Only 12% were able to locate a specific source. When asked directly, 
only 44% were able to articulate how to determine whether on not information 
contained on a website is reliable. These findings suggest the importance of a 
more sustained focus on this skill development area. 
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Table 4: Assessment Questions and Student Scores 



Assessment Question 


Pts. 

Received 


Total 

Possible 

Pts. 


% Pts. 
Received 


How useful do you think this website will be for you in 
gathering information for your research paper? 


232 


406 


57% 


Who is the author and/or sponsor of this website? 


352 


812 


29% 


What is the MAIN purpose of the website? 


690 


1218 


57% 


Why did you choose the answer above (regarding the main 
purpose of the web site)? 


333 


1218 


27% 


Does the information in this website appear to be Opinion or 
Fact? 


705 


1218 


58% 


Why did you choose the answer above (regarding opinion or 
fact)? 


187 


1218 


15% 


Would the information on this website be considered primary 
source, secondary source, or a combination of those? 


101 


406 


25% 


List two sources used in this website 


149 


1218 


12% 


What is the best way to determine whether or not the 
information contained on a website is reliable (trustworthy)? 


90 


203 


44% 


Which of these three sites you have reviewed would be most 
appropriate to use for your assignment? Why? 


66 


203 


33% 



Thus, analysis of the overall results indicate the students in the experimental 
groups outperformed students in the control groups, suggesting the curriculum 
intervention was effective in helping students acquire skills in evaluating 
websites. 

Although encouraging, the results from this pilot study should be viewed 
with some caution in mind. Technically, there was a significant difference 
between the students who received the intervention and those who did not. 
However, several factors must be considered. 

1. Scores increased significantly for the experimental group of students, but 
the increase was slight (2 to 3 points increase). The highest post- 
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assessment mean score was 18.67 out of a total of 40 possible points. 

The scores do not reflect mastery of the content area - there is still a 
great deal of material that students do not grasp. This was reflected both 
quantitatively as well as in the one-on-one post interviews conducted by 
the research team. 

2. The scores of the 9 th grade control group students dropped from 16.06 to 
13.43. One possible explanation for the drop may be the fact that the 
students were given the post-assessment at the very end of the school 
year (June 7 th and June 8 th ) and may have been less invested in their 
performance by the time the test was administered. They may also have 
reacted negatively to being asked to re-take the same assessment after 
receiving no intervening instruction prior to the post-test. 

3. The amount of material covered in the intervention and assessment was 
considerable yet the amount of time teachers spent on the content was 
small - between one and two hours in total. It was not clear from this 
pilot study the potential impacts of a longer intervention period. 

Teachers’ comments were also informative. The 6 th and 9 th grade 
teachers (experimental and control) reported that teaching students to evaluate 
materials on the web is not something they have generally included in their 
curriculum unless a student has questions or asks for help. Both 7 th and 8 th 
grade teachers (experimental and control) indicated that they spend a total of 
about 2.5-3 hours per year providing students with instruction on how to 
evaluate web resources. 

Upon completion of the project, teachers expressed the need to have 
benchmarks for student knowledge of evaluation skills. Teachers revealed that 
the lack of direction for instruction resulted in varying degrees of exposure for 
the students, depending on the teacher’s background and comfort level 
teaching those skills. Benchmarks for student knowledge would allow all 
teachers in the building to better understand specifically what they need to 
teach. 
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Lastly, while teachers supported the intervention and found it well 
designed, their comfort level in using it for the first time was low. They 
suggested that the technology integrationist be more closely involved in the 
initial classroom introduction. 

Conclusions and Recommendations 

The evidence gathered from this project suggests that on the whole, the 
project was successful. MSAD #54 was able to demonstrate that by providing 
students with instruction on how to evaluate digital resources, students did 
improve their skills in evaluating online materials. Thus, it was concluded that 
the pilot study was effective in demonstrating that the intervention could be 
effective in improving students’ 21 st Century Skills. 

Given these preliminary results from this pilot study, several suggestions 
may be gleaned from the study design and findings for future work: 

1 . Extend the length of the intervention . The intervention took place over a 
very short period of time. It is recommended that the time period be 
extended to determine if the extent of the intervention may improve and 
sustain students’ 21 st Century Skills. 

2. Review grade level of materials . The evidence from this pilot study 
indicated the intervention and/or assessments were not effective with 6 th 
grade students. A review of the materials may reveal needed 
modification. 

3. Explore expanding materials . Teachers indicated that the intervention 
materials would be even more useful to them if they were designed 
specifically for each teacher/ level and subject area. 

4. Create a common vocabulary . Teachers felt that common vocabulary 
across all grades for the skills/ terms covered in the intervention would 
be very helpful. 

5. Review the timing of the intervention . Introduce the skills early on in the 
school year so the skills are reinforced as the students engage in 
research activities for different content areas. 



12 




In summary, this pilot study has demonstrated the potential impact of 
interventions specifically designed to address 21 st Century Skills. 

Furthermore, the pilot study has demonstrated the importance and feasibility 
of systematically developing curriculum interventions and collecting and 
analyzing impact data. Further research is encouraged to replicate and 
possibly extend the findings from this pilot study. 

Post Pilot Study Actions 

After completing the work with this project, school officials at SAHS 
began to put into place activities to further their initial work almost 
immediately. Teachers were surveyed in spring 2007 to determine their level of 
instruction around technology. The results were reviewed to distinguish if, 
when, and where technological skills were being taught and to determine if all 
students were receiving equal exposure (Table 5). 



Table 5: Teacher Beliefs Regarding the Need for Teaching of Technological Skills 



19. If you answered ‘Yes’ to Question 18, please choose the Internet skills you would like students to know and use: 


Internet Skills: 


Need 

Students to 
Know 


Do Not Need 
Students to 
Know 


I Do Not 
Know What 
This Is 


Rating 

Average 


Response 

Count 


Ethics and copyrights 


96.4% (54) 


3.6% (2) 


0.0% (0) 


1.04 


56 


Terms: URL, hosts/servers, sites, links, 
hypertext, protocol, download. FTP, 
freeware, shareware 


83.3% (45) 


14.8% (8) 


1.9% (1) 


1.19 


54 


Email: messages, forward, reply to, 
address book, attachments 


90.6% (48) 


9.4% (5) 


0.0% (0) 


1.09 


53 


Web browser: searching, note taking, copy 
& paste, special features 


92.6% (50) 


7.4% (4) 


0.0% (0) 


1.07 


54 


Search Engines: refining searches, types 
and uses 


96.5% (55) 


3.5% (2) 


0.0% (0) 


1.04 


57 


Evaluating and choosing best sites for use 
in given project 


96.4% (54) 


3.6% (2) 


0.0% (0) 


1.04 


56 



A meeting to discuss next steps was held in August 2007 and included 
the English department head, the technology integrationist, the business 
department head, and the building principal. At the meeting, attendees were 
briefed on the intervention structure. Discussions were held to determine the 
most effective way to communicate the intervention information to the entire 
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staff, with the long-term goal of integrating the intervention into their research 
teaching practices. 

A presentation of the survey results and the intervention strategy took 
place at a full faculty meeting on October 1, 2007. The staff was briefed on the 
history of the intervention design as well as the format of the intervention itself. 

The plan is for the intervention to be introduced initially in the Computer 
Applications class, in which all freshmen students are enrolled. They will gain 
the foundations in this setting and the skills will be reinforced in the content 
areas by those teachers. 

The process of ensuring teacher use of the intervention and the 
corresponding student learning will begin with meetings between the 
technology integrationist, English department head, and content area teachers 
of freshmen. SAHS has teams of three teachers of freshmen in the areas of 
English, math, history, and science. Each team has 40 minutes of common 
planning time every other day. Meetings will take place during that planning 
time to discuss the specific research skills and topics that students are 
expected to learn as well as the appropriate time frame for teachers to use the 
intervention. 
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Appendix A 

List of High Schools Participating in the ETS 2006 Early Administration: 



• Canyon Del Oro - AZ 

• Dublin High School * 

• Nichols School - Buffalo, NY 

• Nutley High School - NJ 

• Oak Hill High School - ME 

• Riverside High School * 

• Skowhegan High School - ME 

• Suffern High School - NY 

• Tates Creek High School - KY 

• The exact location of these schools is currently unknown. 
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Appendix B 

Project timeline and tasks 



Skowhegan Project ‘To Do’ List/Timeline 



Task 


Key Participants 


Dates 


V 


1. Develop list of evaluation questions and 
objectives 


Dorothy, Steve, Rick, 
CEPARE 


Jan 




2. Invite content-area teachers to participate 


SAHS & SAMS 
Admin. 


Jan 




3. Create Assessment for pre and post test (timed) to 
measure the evaluation skills covered in the 
intervention (same assessment to be given for the 
pre and post test) 


CEPARE 


Jan/Feb 




a. Select topic and web sites for assessment 
(different topics for pre and post?) 


Dorothy, Steve, Rick, 
CEPARE 


Feb 




b. Pre-test the assessment instrument with a 
few 6 th and 9 th graders. 


Skowhegan teacher 


Feb 




c. Ask for student feedback on what was 
easy, hard or unclear. Review responses 
for possible question revision. 


Dorothy 


Feb 




4. Create documentation log for content-area teacher 
during project 


CEPARE 


Jan 




5. Invite content-area teachers to group meeting to 
provide overview of study and to begin work. 
Items to review include timeframe, 
documentation. Others? Discussion on best way 
to integrate intervention into curriculum in their 
classrooms. 


Dorothy, Steve, Rick, 
CEPARE 


Feb 




6. Some team members complete the assessment in 
order to be able to provide feedback 


Project Team 


Feb 




7. Develop intervention. (Project team and content 
teachers sign off on final intervention 
plan/materials). 


Dorothy, Steve, Rick, 
teachers, CEPARE 


Feb/Mar 




8. Interview participating teachers (control & 
experimental) to obtain background information 
about previous student work around evaluation 
skills as well as about current students in the 
experimental class 


Rick 


Feb 




9. Administer assessment (pre) to 6 th , 8 th and 9 th 
graders 


Skowhegan teachers 


Mar 




a. Administer assessment to experimental 
class 


Skowhegan teachers 


Mar 




b. Administer assessment to control class 


Skowhegan teachers 


Mar 
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10. Develop scoring rubric and score assessments. 
CEP ARE to score assessments and prepare 
summary of results to share with team (but not 
with core-teachers until completion of post- 
assessment). 


CEPARE 


Mar 




11. Pre test Interview (and tape) several students 
from each grade to understand thought processes 
in doing the assessment. 


CEPARE 


April 




a. Need permission slips signed (IRB 
requirement) 


Rick, Dorothy 


Mar 




b. Develop interview guide. 


CEPARE 


Mar 




c. Identify interviewer(s). 


Skowhegan teachers 


Mar 




12. Create post intervention teacher interview 
protocol 


Dorothy, Steve, Rick, 
CEPARE 


Mar 




13. Content Teachers/others at SAHS/SAMS deliver 
intervention. (Content teachers to briefly 
document process each day for each class). 

CEP ARE to observe (and tape?) several sessions. 


Skowhegan teachers, 
Dorothy, Steve, others 


April 




14. Create post intervention Student survey 


CEPARE 


Mar/April 




15. Administer assessment (post) to 6 th , 8 th and 9 th 
graders 


Skowhegan teachers 


May 




a. Administer assessment to experimental 
class 


Skowhegan teachers 


May 




b. Administer assessment to control class 


Skowhegan teachers 


May 




16. Post test Interview (and tape) same students 
from each grade who were interviewed after the 
pre test. To gain an understanding of how their 
thought processes may have changed. 


CEPARE 


May 




17. Conduct post-interview with teachers, both 
experimental & control. 


Rick 


May 




18. CEP ARE to score assessments 


CEPARE 


May 




19. Prepare final report 


CEPARE 


July/Aug 
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Appendix C 

Intervention Teacher Guide and Resources 
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8/26/2007 

To Use a Source or Not Use a Source, That is the 
Question! 

MS AD 54 Evaluation of Resources Guide 

You think you may have found a great site that may provide all the information you 
need in your research. The first thing we want you to remember is to always use more then one 
source in your research. 'Hie second thing we want you to remember is to ask yourself Three 
Essential Questions when you decide to use or not use the resource. This guide will help you 
ask and answer those questions as you make your decision. 

The Three Questions 

Does the content appear to be useful? 

What is the Apparent Purpose? 

How reliable is the information? 





Does the content appear to be useful? 



To determine a website’s usefulness look it over and ask yourself the following; 

• Will this website be useful to my research? 

• Will it help me answer my questions? 

• How easy is it to use this site? 

If you have decided the website provides useful information move on to the question of Purpose. 




What is the Apparent Purpose? 



Understanding Purpose or why the author(s) or sponsor created the site helps you determine how much you can 
rely on the site to give you reliable information. 



1 Study the URL or Uniform Resource locator. This is the address you type in, bookmark, or click on to 
get to a place on the Internet. This is similar to your home address and is where a page or site resides on 
the World Wide Web 



Directories Domain Name 
(folder or file f 0 r site 

names) 



-is a tilde, which 
usually represents 
a personal home 
page 



HyperText Markup 
Language 





suffix or top level 
domain name that 
means government 



Slash represents 
folder in site 



net address 



1 

subdirectory 



name of the web pagej 
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What type of domain does it come from? 

(educational, nonprofit, commercial, government, etc.) 

Is the domain appropriate for the content? 

• Government sites: look for .gov, .mil, .us, or 
other country code 

• Kducational sites: look for edu 

• Nonprofit organizations: look for org 

• If from a foreign country, look at the country 
code and read the page to be sure who 
published it 


8/26/2007 

Look for appropriateness, fit. Wliat kind of information 
source do you think is most reliable for your topic? 


Is it somebody's personal page? 

Read the URL* carefully: Look for a personal name 
(e.g.. jbarker or barker) following a tilde (~ ), a percent 
sign ( % ), or the words "users," "members," or 
"people." 

Is the server a commercial ISP* or 
other provider mostly of web page 
hosting (like aol.com or geocities.com) 


Personal pages arc not necessarily "bad," but you need 
to investigate the author very carefully. For personal 
pages, there is no publisher or domain owner vouching 
for the information in the page. 


Is it published by an source that makes sense? 

Who "published” the page? hi general, the publisher is 
tiie agency or person operating the "server" computer 
from which the document is issued. 

The server is usually named in first portion of the URL 
(between http:// and the first A Have you heard of this 
source before? Does it correspond to the name of the 
site? Should it? 

2. Looking at the author's affiliations and purpose. 


You can rely more on information that is published by 
the source: Look for New York Times news front 
w w w.nvtimes.coni Look for health information from 
any of the agencies of the National Institute of Health 
on sites with nib somewhere in the domain name. 


Who wrote the page? 

Look for the name of the author, or the name of the 
organization, institution, agency, or whatever who is 
responsible for the page 

• An e-mail contact is not enough 

If there is no personal author, look for an agency or 
organization tliat claims responsibility for the page. 

• If you cannot find this, locate the publisher by 
truncating back the URL. Docs this publisher 
claim responsibility for the content? Does it 
explain why the page exists in anyway? 


What are the implications? 

Web pages are all created with a purpose in mind by 
some person or agency or entity. They do not simply 
"grow" on the web like mildew grows in moist comers. 
You are looking for someone who claims accountability 
and responsibility for the content. 

An e-mail address with no additional information about 
the author is not sufficient for assessing the author's 
credentials. 

If this is all you have, try e-mailing the author and 
asking politely for more information about him/her. 


Wliat are the author's credentials on 
this subject? 

Does the purported background or education look like 
someone who is qualified to write on this topic? 


Anyone can put anything on the web for pennies in just 
a few minutes. Your task is to distinguish between the 
reliable and questionable. 


What do others say about the author or responsible 
authoring body? 


"Googling someone" can be revealing. Be sure to 
consider die source. If die viewpoint is radical or 
controversial, expect to find detractors. Think critically 
about all points of view. 
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8/26/2007 




How Reliable Is The Information? 



Is the page dated? Is it current enough? 

Is it "stale" or "dusty" information on a time-sensitive or 
evolving topic? CAUTION: Undated factual or statistical 
information is no better than anonymous information. Don't use 
it. 



How recent the date needs to be depends on your 
needs. For some topics you want current information. 
For others, you want information put on the web near 
the time it became known. In some cases, the 
importance of the date is to tell you whether the page 
author is still maintaining an interest in the page, or 
has abandoned it. 



Primary Vs. Secondary Sources 

Primary' sources are the "materials on a topic upon which 
subsequent interpretations or studies are based, anything from 
firsthand documents such as poems, diaries, court records, and 
interviews to research results generated by experiments, 
surveys, ethnographies, and so on."* 



Primary sources arc records of events as they arc 
first described, without any interpretation or 
commentary. They are also sets of data, such as 
census statistics, which have been tabulated, but not 
interpreted. 

Secondary sources, on the other hand, offer an 
analysis or a restatement of primary sources. They 
often attempt to describe or explain primary sources. 
Some secondary' sources not only analyze primary’ 
sources, but use them to argue a contention or to 
persuade the reader to hold a certain opinion. 





Piim.iiy Source 


Secondary Source 


Art 


Original artwork 


Article critiquing the piece of art 


History 


Slave diary 


Book about the Underground Railroad 


Literature 


Poem 


Treatise on a particular genre of poetry 


Political Science 


Treaty 


Essay on Native American land rights 


Theatre 


Videotape of a performance 


Biography of a playwnght 



Are sources documented with footnotes 
or links? 

• Where did the author get the information? 

As in published scholarly /academic journals 
and books, you should expect 
documentation. 

• If there are links to other pages as sources, are they 
to reliable sources? 

• Do the links work? 



In scholarly/research work, the credibility of most 
writing is proven through footnote documentation or 
other means of revealing the sources of information. 
Saying what you believe without documentation is not 
much better than just expressing an opinion or a point 
of view'. What credibility does your research need? An 
exception can be journalism from highly reputable 
new spapers. But these are not scholarly. Check with 
your instructor before using this type of material. 

Links that don’t w r ork or are to other w r eak or fringe 
pages do not help strengthen the credibility of your 
research. 



If reproduced Information (from another source), is it 
complete, not altered, not fake or forged? 

Is it retyped? If so, it could easily be altered. 

Is it reproduced from another publication? 

Are permissions to reproduce and copyright information 
provided? Is there a reason there are not links to the original 
source if it is online (instead of reproducing it)? 



You may have to find the original to be sure a copy of 
something is not altered and is complete. Look at the 
URL: is it from the original source? If you find a 
legitimate article from a reputable journal or other 
publication, it should be accompanied by the 
copyright statement and/or permission to reprint. If it 
is not, be suspicious. 
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MSAD #54 Website Evaluation Checklist — Teacher Guide 



This well-biown cartoon by Peter Steiner ("On the internet, nobody blows you re a dog. ") 
illustrates an important point relating to online research ; there are a lot of "dogs ’’ out there posing 
as legitimate sources of information. The MSAD # 54 Website Evaluation Checklist is a tool created 
to help students develop strategies for critically evaluating online information resources by asking 
three key questions: 

1 - Does the content of the site appear to be useful? 

2 - What is the site s apparent purpose? 

3 - How reliable is the information? 

This Teacher Guide is an instructional companion piece designed to help you leach the requisite skills. 
An online version of this form and web-based examples may he found at: 




Note: The cartoon is reproduced from the July 5, 1 993 issue of The New Yorker, only for academic discussion, evaluation, and 
research, and complies with federal copyright law as defined and stipulated under Title 17 U. S. Code. 



General Information 

Your Name: 

Research topic/question: 



Quick Scan 

A "Quick Scan " helps students decide, in just a few moments, whether a given website has potential value. A "yes" 
answer here means students should continue the evaluation, and a "no " means they should search for another site. 

Can you read and understand the text on this page? yes | no 

A literacy piece students should be able to read and comprehend at least 85% of the text on a given page. 

If they cannot, then they may miss important context clues that bear on the reliability of the content. 

Does this page contain information you might use? yes | no 

Another literacy piece assuming an adequate level of general comprehension, 
students should be able to scan the text for potentially useful information. 

Is the information current enough for your purposes? yes | no 

Students must first decide whether a given piece of information is time-sensitive, and wlial would be an acceptable 
level of currency. Scan the page for "Last update " information. Other references should note publication dales. 



Site Analysis 

Site Name: 

The site name is NOT the same as the URL. It will tisuallv be found in a banner or heading at the top of the page. 

URL: 

URL stands for "Universal Resource Locator. ” generally blown as a web address. Web addresses themselves contain 
a good bit of useful information regarding the sources of the material and potential biases and agendas (institutional 
or personal) that may impact the overall reliability of the information found on a given site. The diagram on the last 
page describes in more detail how to "read" a web address. 

Domain name: 

Domain names are usually chosen only for their name recognition value. It is the first thing a search engine scans for. 
Site developers acquire domain names by simply buying the rights to them, often trapping the unwary web surfer with 
a benign-sounding URL. In one of the more egregious examples of this ploy, the domain name "marlinlulherking.org" 
is owned by a white supremacist group. 

Extension: 

Extensions provide clues about the organizational roots of a website: government-sponsored sites contain .gov. 
military sites contain .mil. educational institutions .edu, non-profit organizations .org. and soon. The chart on the last 
page contains a more comprehensive list of extensions and their common affiliations. 
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Page name: 

On most websites, the page name appears at the very toj) of the browser window. 

It will also appear as the very last item in the URL, typically followed by the suffix html or htm. 

Site sponsor and/or author: 

The sponsor may be identified in the domain name itself or elsewhere on the first (index) page. The author may be 
solely responsible for the content, or may be an agent of the sponsoring organization. Information from established, 
responsible, institutiotud sources is generally reliable, but may reflect institutional biases. Be wary of the "sponsored 
links ” that appear prominently on most search result pages. These almost always exist to sell you something. 

Is this a personal page? yes | no | unsure 

As the diagram illustrates, a tilde (~) is a good indicator you 're looking at a personal page. Other clues include 
percent signs (%) and such words as “users, ” “ members , " " people . ” and so on. Using information from a persona! 
page is not necessarily a bad thing, but you need to examine the content carefully. It s wise to research the author s 
credentials and to cross-check personal page information with other resources. 

Explain your reasoning: 

How can you justify this claim ? If hat cues can you find in the URL or elsewhere on the page ? 

The main purpose of this website is to: 

supply information | provide a service | state an opinion | sell a product entertain | unsure 

Most websites do have a primary focus, and information found there will be presented irt ways that support that focus. 
Since tire purpose of online research is to obtain information, websites whose primary focus is something other than 
supplying information must he more carefully screened for biases that reflect that focus. Consumer Reports and the 
Ford Motor Company may both have similar information on the latest Ford pickup trucks, but their slants will differ. 
Explain your reasoning: 

How cimyou justify this claim? What cues can you find in the URL or elsewhere on the page ? 



+ Content Analysis 

Given the results of your site analysis, how might this information be biased? 

Unless we ‘re dealing with simple facts ( When is your birthday? What’s the chemical formula for water?) we can 
assume that most of what we read is biased to some extent. If students are aware of the mechanisms through which the 
information they are seeking has been filtered, they 'll make better judgments about its usefidness for their purposes. 

The relevant information on this site is: primary source secondary source | both 

A primary source is a document which describes an event by its witnesses or first recorders. Some types are: diaries, 
speeches, letters, interviews, newspapers, autobiographies and official records including government publications , 
legislation, court reports, etc. Secondary sources are one step removed from the event being described, but can 
provide background for or clarification of primary sources. Some types are: textbooks, journal articles, 
histories, criticisms, commentaries, and encyclopedias. 

Are additional sources of information provided? yes | no 

These may consist of links to other online resources or bibliographic references to print materials. Their very 
presence on the page indicates that the author is making some attempt to validate his/her claims. You should examine 
these to verify that links work and that other reference materials do in fact exist. 

If “yes,” list two or three sources you might use. 

The number atid type of referenced resources your students consult depends on the degree of rigor you expect, but 
students shotdd definitely check the content of at least two of these sources to see if they truly support claims made 
on the original site. 

How much useful information did you find here? just a little | quite a bit | all I need 

This depends largely on the type of information sought. Answers to the question, “Why did Hitler want to 
eradicate the Jews? ’’ will require more in-depth analysis than “Haw many Jews died in the Holocaust? " 

Rate the quality of the information on this site: poor adequate | good | excellent 

The information gathered to this point in the evaluation provides the raw material for making these judgements. 

Is the originating domain one that is likely to contain good information on topics of this sort? Is the site sponsor 
generally reputable? Do the author 's credentials inspire confidence? Is the writing clear and unambiguous? Is 
there evidence of bias? Is the information verifiable? Is it current enough to be useful? Do referenced materials 
support the author 's claims? Are these materials themselves credible? If all of this checks out, you ve probably found 
a reliable resource. 
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Reading a URL 



Directories (slashes repre- Domain name 
sent folders] or file names) for site 



HyperText 

Folder or page name. A tilde (~) Markup 
usually represents a personal page. Language 



http://www.infosite 



org/^jdoe/topic.html 



HyperText World Wide 
Transfer Web 

Protocol 



Domain Web page 

name name 

extension 



Truncating a URL can provide insights into the nature of the entity sponsoring a website. 
To do this, simply delete all text following the domain name extension and reload the 
page. In the example shown above, the truncated URL would read: 

http://www.infosite.org/ 



Common Domain Name Extensions 

For a more comprehensive list, including country extensions, visit: 

http :// www. c omp uteruser.co m/re sources/dictio nary/n o tram es/nf. domains, html 

.ac — educational network (same as .edu) 

.biz — business 

.com — commercial site in the US 

.edu — educational site in the US 

.firm — business 

.gov — U.S. government 

.int — international institution 

.mil — U.S. military 

.nato — NATO site 

.net — administrative site in the US 

.nom — personal site 

.org — organization in the US 

.store — retail business 

.web — about the World Wide Web 
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MSAD #54 Website Evaluation Checklist 



Instructions: Circle the appropiate response where choices are provided. Answer all other questions as best you can. In the Quick Scan 
section, a "yes" answer means you shotdd continue with this evaluation, and a “ no ” means you should search for a more useful website. 



General Information 

Your Name: 

Research topic/question: 



^ Quick Scan 

Can you read and understand the text on this page? 
Does this page contain information you might use? 

Is the information current enough for your purposes? 



yes | no 
yes | no 
yes | no 



Site Analysis 

Site Name: 

URL: 

Domain name: 

Extension: 

Page name: 

Site sponsor and/or author: 

Is this a personal page? yes | no | unsure 

Explain your reasoning: 

lire main purpose of this website is to: 

supply information | provide a service | state an opinion sell a product | entertain | unsure 
Explain your reasoning: 



we~ Content Analysis 

Given the results of your site analysis, how might this information be biased? 



The relevant information on this site is: primary source | secondary source both 

Are additional sources of information provided? yes | no 

If “yes,” list two or three sources you might use. 



How much useful information did you find here? just a little | quite a bit | all I need 

Rate the quality of the information on this site: poor [ adequate | good | excellent 
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Appendix D 

Pre-and Post Assessment 



ICT Research Project 

Untitled Page 



Dear Student: Skowhegan Area Elementary, Middle School and High School are working to improve your 21st 
Century Skills. In order to strengthen your Information Communication Technology (ICT) Literacy skills, we 
have asked the Center for Education Policy, Applied Research, and Evaluation (CEPARE) at the University of 
Southern Maine to help create a project that will offer you the chance to improve your online research skills. 

The following assessment will help us to understand your abilities in the area of evaluating the usefulness of 
online information. We will ask you to answer all the questions as they relate to each of three websites 
provided. None of the information collected here will in any way affect your grade or your relationship with 
the Skowhegan school system. You are not required to participate in this study if you do not wish to. Your 
honest responses to the questions on this assessment will be helpful and provide you the opportunity to 
contribute to the current knowledge about the process for teaching evaluation skills using digital media and 
in so doing effect possible positive change to classrooms throughout MSAD 54. 

There is no direct foreseeable risk for participating in this study. Your individual responses will be seen only 
by the evaluation team at CEPARE. Involvement in this study is completely voluntary. Reports will not include 
any information that will make it possible to identify a participant. In any sort of quotation we will use, we 
will not include any information that will make it possible to identify the student being quoted. 

If you have any questions about this project you may reach the CEPARE evaluation team at (207)780-5044 
or by email at cepare@usm.maine.edu. If you have questions about your rights as a research subject, you 
may contact: Director, Office of Research Compliance, University of Southern Maine at (207)780-4268 or by 
email at usmirb@maine.edu. 

Thank you for being willing to participate in this important project. 

Sincerely, 

Dorothy Small 
Superintendent, MSAD 54 



27 







ICT Research Project 

Untitled Page 



Please enter you name, today's date, grade level, and teacher's name in the spaces provided. 



First and Last Name 



Please enter today's date (for example, 3/12/07) 

MM DD YYYY 

Today"s Date | / I — /I 



What grade are you in? 



Teacher name 



ICT Research Project 



Scenario: You have been assigned an Internet research project. The project is about nutrition, and your 
assignment is to create a 7-day menu of nutritious, balanced meals. You will use information that you collect 
from the Internet to create your menu. 

Your task today: Look at each of three websites (one at a time) and answer the survey questions about 
each site. You will be asked to evaluate each site and determine how useful it will be in creating your 7-day 
menu. 

You will need to answer the survey questions and look at the website at the same time. To do this, simply 
click on a website to open a new window. This will allow you to look at the survey questions in one window 
and the website you are evaluating in another window. 

You may need to resize and/or move both windows in order to look at them at the same time. If you have 
problems looking at the website and the survey at the same time, please ask your teacher for assistance. 
And remember, DO NOT CLOSE THE SURVEY WINDOW AT ANY POINT OR YOUR ANSWERS WILL BE LOST. You 
may, however, close the different website windows. You must answer all of the questions on each page in 
order to move to the next page. 
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no *" * ooooo « !° ooo 



ICT Research Project 

Website #1 



Website #1 - The Eoo Nutrition Center 
click above link to access site 

Remember: your assignment is to determine if this website would be a good source of information for your 
project creating a 7-day menu of nutritious, balanced meals. 



1. How useful do you think this website will be for you in gathering information 
for your research paper? 

None or very little of It Is relevant (useful) to my topic 
Some of it is relevant (useful) to my topic 
All or almost all of it is relevant (useful) 



Who is the author and/or sponsor of this website? 



What is the MAIN purpose of the website? Are the authors trying to: 

Inform the reader 
Persuade the reader 
Entertain the reader 
Sell something to the reader 
Other (please specify) 



Why did you choose the answer above? 



£l 

Does the information in this website appear to be Opinion or Fact? 

All opinion 

Mostly opinion and some fact 
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ICT Research Project 



Mostly fact and some opinion 
Q All fact 

6. Why did you choose the answer above? 



7. How current (up-to-date) is the information contained on this website? 

Most/all of it is current 
Some of it is current 
Not current but still usable 
Not current and not usable 
Do not know 

8. Would the information on this website be considered primary source, 
secondary source or a combination of those? 

Primary source 
Secondary source 

Combination of primary and secondary sources 
Do not know 

9. List two sources used in this website (if you do not know what the sources 
are, please type "do not know" in the space provided). 

a) 
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ICT Research Project 

Website #2 



Website #2 - Shape Uo America! 

close website #1 and click above link to access website #2 

Remember: your assignment is to determine if this website would be a good source of information for your 
project creating a 7-day menu of nutritious, balanced meals. 

10. How useful do you think this website will be for you in gathering information 
for your research paper? 

None or very little of it is relevant (useful) to my topic 
Some of it is relevant (useful) to my topic 
Ail or almost all of it is relevant (useful) 

11. Who is the author and/or sponsor of this website? 



12. What is the MAIN purpose of the website? Are the authors trying to: 

Inform the reader 
Persuade the reader 
(^) Entertain the reader 

Sell something to the reader 
Other (please specify) 



13. Why did you choose the answer above? 

1 

£ 

14. Does the information in this website appear to be Opinion or Fact? 

All opinion 

Mostly opinion and some fact 
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ICT Research Project 



Mostly fact and some opinion 
Q All fact 

15. Why did you choose the answer above? 



16. How current (up-to-date) is the information contained on this website? 

Most/all of it is current 
Some of it is current 
Not current but still usable 
Not current and not usable 
Do not know 

17. Would the information on this website be considered primary source, 
secondary source or a combination of those? 

Primary source 
Secondary source 

Combination of primary and secondary sources 
Do not know 

18. List two sources used in this website (if you do not know what the sources 
are, please type "do not know" in the space provided). 

a) 
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ICT Research Project 

Website #3 



Website #3 - Nutrition for a Living Planet 

close website #2 and click above link to access website #3 

Remember: your assignment is to determine if this website would be a good source of information for your 
project creating a 7-day menu of nutritious, balanced meals. 

19. How useful do you think this website will be for you in gathering information 
for your research paper? 

None or very little of It is relevant (useful) to my topic 
Some of it is relevant (useful) to my topic 
All or almost all of it is relevant (useful) 

20. Who is the author and/or sponsor of this website? 



21. What is the MAIN purpose of the website? Are the authors trying to: 

Q ^ Inform the reader 
Persuade the reader 
(^) Entertain the reader 

Sell something to the reader 
Other (please specify) 



22. Why did you choose the answer above? 

1 

£ 

23. Does the information in this website appear to be Opinion or Fact? 

All opinion 

Mostly opinion and some fact 
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ICT Research Project 



Mostly fact and some opinion 
Q All fact 

24. Why did you choose the answer above? 



25. How current (up-to-date) is the information contained on this website? 

Most/all of it is current 
Some of it is current 
Not current but still usable 
Not current and not usable 
Do not know 

26. Would the information on this website be considered primary source, 
secondary source or a combination of those? 

Primary source 
Secondary source 

Combination of primary and secondary sources 
Do not know 

27. List two sources used in this website (if you do not know what the sources 
are, please type "do not know" in the space provided). 

a) 
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ICT Research Project 

Comparison 



Please open each of the websites you have looked at (linked below). The computer will allow you to have all 
of the websites open at the same time as well as the survey. Look at each of the websites, either 
separately or all together, one more time and consider the following questions. 

The Egg Nutrition Center 

. Shape . U p, A m eri ca !, 

28. What is the best way to determine whether or not the information contained 
on a website is reliable (trustworthy)? 



29. Which of the three sites you have reviewed would be most appropriate to 
use for your assignment? Why? 
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Appendix B 

Websites used for pre-and post-assessments 

Website #1: The Egg Nutrition Center 
http:/ / WAVw.enc-online.org/ 

Website #2: Shape Up America! 
http: / / www.shapeup.org/ 

Website #3: Nutrition for a Living Planet 
http:/ /www.diet-and-health.net/ 
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Appendix F 

Student Interview Questions (pre & post) 

1 . How easy or hard did you find the survey? Was it easy, hard, or in the 
middle? [Probe] 

2. You said that (none, some, all) of the information on this 

website was relevant or useful to your topic. What did you see that you felt 
was useful? What did you see that you felt wasn’t going to be useful? 

3. You indicated that the MAIN purpose of the website was 

(persuade, inform, sell). Can you tell me a little bit more about that? 

4. You said that the information on the website was (fact/ opinion). 

Can you tell me more about that? 

5. On the survey you indicated that (most/ some /not current by 

usable / unusable) . How did you know? 

6. You said the information on the site was (primary, secondary 

source) . What specifically did you see on the site that indicated it was a 
primary/ secondary source? 

Repeat questions 1-6 for each of the three websites. 

7. When you use the Internet to do research for a class assignment, what 
steps do you take to find information to use for your assignment? (Have 
student walk you through the process they use) . 

8. Imagine that you went to the Internet and found a really good site for your 
project - you felt like you hit the jackpot. Tell me the characteristics that a site 
would have that would make it so perfect. 

9. Often times when you use a search engine, many, many sites come up. 

How do you know which ones are worth looking at? 

10. How do you know if the information you find is reliable or truthful? 

[Probe] Are there any other ways you can tell? 
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Appendix G 

Teacher Interview Questions (pre) 



1. Years of teaching experience: 

a. Total years 

b. Years teaching current subject 

2. How much time have you spent on teaching evaluation skills (per year) prior 
to this study? (provide the list of skills we’re covering - determining 
usefulness, reliability, etc.) 

3. What strategies have you used in the past to teach students evaluation 
skills? What has worked well, not so well? Please describe. 

4. In the past, what obstacles have you seen students face when confronted 
with the task of evaluating websites when doing research on the Internet? 

5. Overall, what are your impressions of students’ ability to evaluate sources 
at the beginning of the year well as at the end of the year? In other words, in 
your experience, how do students’ evaluation skills change during grade (6), 

(8), (9)? 



Teacher Interview Questions (post) 

1 . How much time (hours) did you spend doing the intervention with your 
students? 

2. What were your general impressions of the intervention? 

a. What are the strengths of the intervention? 

b. What could be improved about the intervention? 

3. Were there any surprises for you throughout the process of this project? 

4. Using a scale of 1 to 10 where 1 is “not at all knowledgeable” and 10 is 
“completely competent”, how would you rate your students’ skill level in 
evaluating websites prior to the intervention vs. post-intervention? 

a. If there was a lot of movement - why? 

b. If there was little movement - why? 

5. If there are students who still need more work developing these skills, what 
do you feel they need? (What can be done to help them further?) 

6. How good of a match do you think there was between the intervention and 
the assessment? Why do you say that (please be as specific as possible)? 
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Appendix H 



ICT Research Project - Post Assessment 
Scoring Rubric 



Website #1 

1. How useful do you think this website will be for you in gathering information for your 
research paper? 



1 point 


0 points 


• Some of it is relevant (useful) to my topic 


• None or very little of it is relevant (useful) 
to my topic 

• All or almost all of it is relevant (useful) 



2. Who is the author and/or sponsor of this website? 



1 point 


0 points 


• Egg Nutrition Center 


• Anything else 



3. What is the MAIN purpose of the website? Are the authors trying to: 



2 points 


1 point 


0 points 


• Persuade the reader 


• Inform the reader 


• Entertain the reader 

• Sell something to the 
reader 

• Other 


4. Why did you choose the answer above? (Examples of responses) 


2 points 


1 point 


0 points 


• Because the author 

attempts to persuade the 
reader into believing 
eggs arc nutritious, 
delicious, & affordable 


• Website states that its 
target audience is egg 
lovers, egg producers/ 
processors, and health 
care providers who want 
to learn more about how 
eggs contribute to a 
healthy diet 


• Because this website is 
for egg lovers and it is 
supposed to entertain 
them 


5. Does the information in this website appear to be Opinion or Fact? 


2 points 


1 point 


0 points 


• Mostly fact and some 
opinion 


• Mostly opinion and some 
fact 


• All opinion 

• All fact 



39 







6. Why did you choose the answer above? (Examples of responses) 



2 points 


1 point 


0 points 


• Because they are saying 
things that are true, but 
they also say what they 
think about the eggs 


• Because it mostly states 
what they think of eggs. 
The other part is fact 
because they’re trying to 
give you information on 
the subject so that you’ll 
get an interest and join 
their site 


• Because there aren’t any 
facts on this page 

• Because it has no 
opinions 



8. Would the information on this website be considered primary source, secondary source 
or a combination of those? 



1 point 


0 points 


• Combination of primary & secondary 
sources 


• Primary source 

• Secondary source 

• Other/ do not know 



9. List two sources used in this website 

• 1 point for each listed credible source 

Website #2 

10. How useful do you think this website will be for you in gathering information for your 
research paper? 



1 point 


0 points 


• Some of it is relevant (useful) to my topic 


• None or very little of it is relevant (useful) 
to my topic 

• All or almost all of it is relevant (useful) 



11. Who is the author and/or sponsor of this website? 



1 point 


0 points 


• Shape Up America 

• C Everett Koop 


• Anything else 



12. What is the MAIN purpose of the website? Are the authors trying to: 



2 points 


1 point 


0 points 


• Inform the reader 


• Persuade the reader 


• Entertain the reader 

• Sell something to the 
reader 

• Other 



13. Why did you choose the answer above? (Examples of responses) 
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2 points 


1 point 


0 points 


• They are frying to inform 
the reader of obesity as a 
health issue and to 
provide responsible 
information on healthy 
weight management 


• Trying to persuade the 
reader to lose more 
weight and eat right 


• Because they are trying to 
sell a cookbook 



14. Does the information in this website appear to be Opinion or Fact? 



2 points 


1 point 


0 points 


• Mostly fact and some 
opinion 


• Mostly opinion and some 
fact 


• All opinion 

• All fact 



15. Why did you choose the answer above? (Examples of responses) 



2 points 


1 point 


0 points 


• The information 
presented appears 
factual, though not much 
of it cites source 
material... any opinions 
expressed on this site are 
very closely tied to this 
(apparently) factual 
information 


• Because they are trying 
to tell you what diets arc 
the most healthy, and 
they have surveys as fact 
to prove it 


• Because it’s all opinion 

• Based on fact with some 
generalizations 



18. List two sources used in this website 

• 1 point for each listed credible source 

Website #3 

20. Who is the author and/or sponsor of this website? 



2 points 


1 point 


0 points 


• Author/ sponsor is not 
listed 


• DietandHealth.Net 


• Other 



21. What is the MAIN purpose of the website? Are the authors trying to: 



2 points 


1 point 


0 points 


• Inform the reader 


• Persuade the reader 


• Entertain the reader 

• Sell something to the 
reader 

• Other 



22. Why did you choose the answer above? (Examples of responses) 
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2 points 


1 point 


0 points 


• Because they are 
informing you about 
what you can do to keep 
yourself healthy 


• To persuade the reader to 
make us eat better 


• They are trying to get you 
to think their product is 
good 


23. Does the information in this website appear to be Opinion or Fact? 


2 points 


1 point 


0 points 


• Mostly fact and some 
opinion 


• Mostly opinion and some 
fact 


• All opinion 

• All fact 


24. Why did you choose the answer above? 


2 points 


1 point 


0 points 


• There arc facts about 
what you can do to stay 
healthy, and there is 
opinion about what foods 
and exercises are most 
effective 


• Because they have things 
that are suggested, which 
means that it isn’t 
complete fact, with 
mostly opinions 


• I choose that because it 
seems like all fact 

• I think it is all opinion 
because people were 
telling you things from 
their point of view 



26. Would the information on this website be considered primary source, secondary source 
or a combination of those? 



1 point 


0 points 


• Combination of primary & secondary 
sources 


• Primary source 

• Secondary source 

• Other/ do not know 



27. List two sources used in this website 

• 1 point for each listed credible source 

Comparisons 

28. What is the best way to determine whether or not the information contained on a 
website is reliable (trustworthy)? 

• 1 point for at least 1 credible method 



29. Which of these three sites you have reviewed would be most appropriate to use for 
your assignment? Why? 

• 1 point for listing a site & a credible reason for selecting that site 
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